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Summary 


Problem  and  Background 

The  challenges  facing  Antisubmarine  Warfare  (ASW)  training  in  the  late  1990s  are 
greater  than  at  any  time  since  the  early  days  of  World  War  II.  Conditions  since  the  end 
of  the  Cold  War,  and  those  expected  throughout  the  next  decade,  impose  additional 
complexity  on  maintaining  U.S.  ASW  superiority.  Russian  nuclear  submarine 
technology  continues  to  improve  and  advanced  submarines  continue  to  be  built  and 
delivered  to  their  fleet.  Concurrently,  the  proliferation  of  improved  diesel  submarine 
technology  to  many  Third  World  nations  requires  that  our  ASW  forces  also  be  capable  of 
conducting  operations  in  the  vastly  different  littoral  regions. 

The  training  challenge  is  two-fold:  (1)  retaining  the  capability  to  detect  and 
prosecute  nuclear  submarines,  and  (2)  expanding  our  current  capability  against  diesel 
submarines  of  the  Third  World.  When  coupled  with  dramatic  reductions  in  ASW 
training  resources,  including  at-sea  training,  this  historic  change  compels  the 
development  of  training  for  skills  learned  previously  on  the  job  and  for  skills  required  in 
new  environments. 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  System  was  developed  to 
address  post  Cold  War  ASW  training  requirements.  Specifically,  IMAT  is  designed  to 
teach  the  complex  conceptual  knowledge  and  cognitive  and  procedural  skills  required  to 
reason  about  the  interrelationships  among  the  operating  modes  of  target  submarines,  the 
environmental  variables  that  affect  sound  transmission,  and  the  sensor  systems  used  for 
detection  and  localization.  In  addition,  IMAT  provides  extensive  training  on  submarine 
acoustics  and  interpretation  of  sensor  system  data.  The  IMAT  approach  to  training  is 
based  on  recent  advances  in  cognitive  psychology  and  instructional  technology.  It 
combines  cognitive  analytic  and  curriculum  design  technology  with  advanced  computer- 
based  graphics  (scientific  visualization)  and  programming  technology  to  present  state-of- 
the-art  training. 

Objective 

The  objective  of  this  effort  is  to  evaluate  the  application  of  the  IMAT  system  in  the 
acoustic  analysis  portion  of  the  Aviation  Systems  Warfare  Operator  (AW)  “A.” 

Method 

The  AW  “A”  Acoustic  Analysis  lessons  were  evaluated  on  (1)  student  performance 
on  two  types  of  end  of  unit  test  items  (fact  items  and  cognitive  skill/acoustic  analysis 
items),  (2)  student  motivation,  and  (3)  quality  of  instructional  design.  Fifty-three 
students,  trained  with  the  IMAT  system,  were  compared  with  22  students  trained  with  the 
standard  instruction  used  in  the  “A”  school  prior  to  the  introduction  of  IMAT. 


Results 


MAT  students  performed  better  than  standard  instruction  students  on  both  fact  items 
and  cognitive  skill  items.  This  finding  shows  that  the  combination  of  strategies 
incorporated  in  the  MAT  system  provides  an  effective  environment  for  learning 
complex  knowledge  and  skills.  Further,  MAT  students  performed  better  on  cognitive 
skill  items  relative  to  fact  items  than  the  standard  instruction  students.  This  result  is 
exactly  what  MAT  was  designed  to  accomplish  and  directly  addresses  the  post  Cold  War 
training  requirements  for  sensor  operators. 

The  instructional  design  analysis  found  that  in  the  MLAT  lessons  all  required 
instructional  components  were  present,  while  components  were  present  to  a  lesser  degree 
for  the  standard  instruction  lessons. 

The  motivation  analysis  showed  that  MAT  instruction  compared  favorably  with 
similar  computer  based  instruction  and  was  generally  more  motivating  than  standard 
classroom  instruction. 

Conclusions 

Several  conclusions  can  be  drawn  from  this  study.  First,  the  MAT  approach  to 
classroom  training,  which  is  based  on  computer-generated  dynamic  displays  (scientific 
visualization)  and  instructional  design  and  delivery  strategies  designed  to  enhance 
cognition,  produced  substantial  gains  in  performance.  This  is  especially  true  for  the 
explicit  MAT  goal  of  teaching  complex  knowledge  and  cognitive  skills.  Second,  the 
MAT  system  emphasis  on  adhering  to  the  principles  of  high  quality  instructional  design 
was  supported  in  the  instructional  design  analyses.  The  differences  in  design  quality 
between  the  standard  instruction  and  MAT  groups  undoubtedly  contributed  to  the 
observed  performance  differences.  Overall,  the  results  show  that  MAT  is  a  highly 
effective  training  system  that  offers  a  viable  solution  for  many  of  the  training 
requirements  and  challenges  faced  in  the  post  Cold  War  world. 
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Introduction 


Problem  and  Background 

The  challenges  facing  Antisubmarine  Warfare  (ASW)  training  in  the  late  1990s  are 
greater  than  at  any  time  since  the  early  days  of  World  War  II.  During  the  Cold  War, 
ASW  mission  requirements  (and  the  training  designed  to  support  those  requirements) 
were  driven  by  the  need  to  combat  Soviet  nuclear  submarines  in  open  ocean 
environments.  The  prevailing  ASW  strategy  was  designed  to  detect  and  prosecute  enemy 
submarines  at  long  ranges,  and  operations  were  most  often  conducted  by  single  ASW 
units  with  minimal  coordination  with  outside  assets.  The  relatively  benign  deep  ocean 
environment  and  nearly  exclusive  focus  on  the  Soviet  threat  resulted  in  the  development 
of  effective  and  well  practiced  sensor  and  weapons  tactics.  ASW  training  mirrored  that 
relatively  narrow  focus,  and,  combined  with  frequent  real-world  encounters,  was  quite 
effective  in  producing  competent  ASW  sensor  operators. 

Conditions  since  the  end  of  the  Cold  War,  and  those  expected  throughout  the  next 
decade,  impose  additional  complexity  on  maintaining  our  ASW  superiority.  Russian 
nuclear  submarine  technology  continues  to  improve  and  advanced  submarines  continue 
to  be  built  and  delivered  to  their  fleet.  Concurrently,  the  proliferation  of  improved  diesel 
submarine  technology  to  many  Third  World  nations  requires  that  our  ASW  forces  also  be 
capable  of  conducting  operations  in  the  vastly  different  littoral  regions. 

Littoral  environments  introduce  added  difficulty  in  optimally  employing  onboard 
sensors  and  weapons  tactics.  Coastal  areas  with  shallow  water,  complex  bathymetry  and 
bottom  topography,  heavy  shipping,  and  highly  variable  environmental  conditions 
impose  significant  restrictions  on  traditional  deep  water  tactics.  The  diesel  submarine 
operating  in  home  waters  has  the  additional  advantage  of  familiarity  with  environmental 
anomalies,  slow  speed  operation,  and  support  from  own  nation  defense  systems. 

The  training  challenge  has  thus  become  two-fold;  retaining  the  Cold  War  capability 
to  detect  and  prosecute  nuclear  submarines  while  expanding  our  current  capability 
against  diesel  submarines  of  the  Third  World.  When  coupled  with  dramatic  reductions  in 
ASW  training  resources,  including  at-sea  training  opportunities,  this  historic  change 
compels  the  development  of  different  approaches  to  training  for  sensor  operators. 

Traditional  ASW  Training  for  Sensor  Operators 

In  the  past  the  ASW  training  has  been  based  on  a  balance  between  schoolhouse  and 
operational  training.  To  become  journey  level  operators,  students  learned  the  basics  in 
schoolhouse  training  and  then  received  extensive  on-the-job  training  through  supervised 
practice  by  experienced  fleet  crews  at-sea.  The  schoolhouse  training  focused  on 
memorizing  facts,  procedures,  and  large  databases  of  threat  intelligence  parameters. 
Students  were  not  taught  to  think  about  and  relate  the  underlying  physics  of  threats,  the 
environment  and  the  sensor  systems.  This  approach  resulted  in  graduates  who  could 
answer  specific  factual  questions  based  on  memorized  information  and  who  could 
perform  procedures  but  who  had  difficulty  in  applying  knowledge  and  principles  to  solve 
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problems  in  operational  situations.  Great  reliance  was  placed  on  a  substantial  amount  of 
at-sea  experience  to  transfer  the  knowledge  and  skill  gained  in  formal  training  to 
operational  competency. 

Post  Cold  War  ASW  Training  Requirements 

All  four  ASW  training  communities  (surface,  subsurface,  air,  and  surveillance)  now 
agree  that  the  combined  change  in  threat,  environment,  and  operating  circumstances 
requires  a  new  training  approach. 

Current  fleet  ASW  practitioners  receive  little  at-sea  operational  practice  against  non- 
cooperative  submarines.  As  a  result,  an  operator’s  ability  to  detect,  classify,  and  track 
submarine  targets  will  be  directly  related  to  the  quality  of  initial  training,  and  to  how  well 
those  skills  are  maintained  and  increased  throughout  a  career.  To  achieve  this  level  of 
competence,  the  limited  hours  available  for  both  schoolhouse  and  at-sea  training  must  be 
used  to  provide  both  a  solid  conceptual  understanding  of  the  complex  tasks  that  operators 
must  perform  as  well  as  the  opportunity  to  practice  those  tasks  under  varying  real-world 
conditions. 

Achieving  these  goals  will  require  substantive  changes  the  execution  of  schoolhouse 
and  operational  training.  ASW  training  must  see  dramatic  modifications  to  passive 
acoustic  analysis  training  and  substantial  expansion  of  training  in  active  sonar,  radar,  and 
electromagnetic  sensor  systems  to  enable  operators  to  contribute  effectively  in  a  multi¬ 
sensor  approach  to  submarine  prosecution.  Further,  the  complexity  of  the  acoustic 
environment  in  the  littoral  regions  requires  a  substantial  increase  in  knowledge  of  the 
effects  of  ocean  bathymetry  on  acoustic  energy  transmission  and  how  that,  in  turn, 
affects  sensor  selection  and  placement.  These  training  requirements  can  only  be  met 
through  principle-based  application  of  training  technologies,  which  can  provide 
conceptual  knowledge  and  high  fidelity  experience  to  offset  the  lack  of  at-sea  practice. 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  System 

Overview 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  System  was  developed  to 
address  post  Cold  War  ASW  training  requirements.  Specifically,  IMAT  is  a  classroom 
based  approach  to  training  that  is  designed  to  teach  the  complex  conceptual  knowledge 
and  cognitive  and  procedural  skills  required  to  reason  about  the  interrelationships  among 
the  operating  modes  of  target  submarines,  the  environmental  variables  that  affect  sound 
transmission,  and  the  sensor  systems  used  for  detection  and  localization.  In  addition, 
IMAT  provides  extensive  training  on  submarine  acoustics  and  interpretation  of  sensor 
system  data. 

The  IMAT  approach  to  training  is  based  on  recent  advances  in  cognitive  psychology 
and  instructional  technology  such  as  scientific  visualization  (Back  &  Layne,  1988, 
Bryson,  1994)  and  anchored  instruction  (Cognition  &  Technology  Group  at  Vanderbilt, 
1990).  It  combines  cognitive  analytic  and  instructional  design  technology  with  advanced 
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computer-based  graphics  and  programming  technology.  The  result  is  scientific 
visualization,  a  dynamic  graphical  interface  integrated  with  state-of-the-art  instructor  and 
student  guides  which  provides  the  traditional  classroom  instructor  with  a  capability  to 
effectively  teach  complex  cognitive  concepts  and  skills.  In  the  past,  achieving  this 
capability  has  been  hampered  by  limitations  in  cognitive  task  analysis,  and  particularly  in 
cognitive  models.  In  addition,  limitations  in  computer  capabilities  have  precluded  the 
development  of  cause  and  effect  representations  of  highly  complex,  multi-modal  tasks 
that  are  required  of  expert  practitioners.  Recent  developments  in  cognitively-based 
training  design  have  demonstrated  that  models  of  physical  phenomena  can  be  integrated 
with  high  resolution  graphics  to  demonstrate  the  interactive  relationships  (Bryson,  1994). 
The  IMAT  system  extends  that  technology  in  the  traditional  classroom  environment,  with 
specific  emphasis  on  cognitive  design  models  that  account  the  knowledge  structure 
interrelationships  of  threat,  environment,  and  system  for  operator  training.  IMAT 
provides  cause  and  effect  training  for  decision  making  with  multiple  system  input,  and 
allows  visualization  of  interactive  spatial  relationships  among  operators,  sensors  and 
other  platforms. 

Specifically,  for  sensor  operators  IMAT  apprentice  training  is  presented  in  a  mission 
context  with  a  substantial  emphasis  on  the  interactive  relationship  among  environmental 
factors,  threat  behavior,  and  sensoi;  system  capabilities  and  constraints.  Because  sensor 
operators  at  the  apprentice  level  have  limited  backgrounds  in  submarine  operations,  the 
physics  underlying  energy  transmission,  and  in  operating  complex  sensor  systems, 
establishing  a  good  conceptual  understanding  of  the  complicated  interactions  that  occur 
in  the  real  world  is  the  essential  foundation  for  learning  effective  sensor  operation. 
Students  are  taught  to  understand  that  the  enemy  submarine’s  mission  will  largely  dictate 
its  operating  mode  including  course,  speed,  and  depth  of  operation,  and  that  the  operating 
mode  of  the  submarine  defines  its  vulnerabilities  to  onboard  acoustic  and  electromagnetic 
sensors.  They  are  further  taught  how  the  relative  complexity  of  the  ocean  environment 
will  impact  detection  ranges,  search  rates,  and  contact  duration.  Smdents  learn  to  reason 
through  these  interactions  in  a  cause  and  effect  learning  process.  Multi-dimensional 
interactions  are  displayed  visually  and  the  IMAT  instruction  provides  qualitative 
explanations  for  the  interactions  that  occur.  IMAT  presents  demonstrations  of  varying 
outcomes  based  upon  changes  in  the  threat  or  environment  to  promote  the  development 
of  the  principle-based  knowledge  critical  for  adaptations  to  the  variations  presented  in 
real  world  situations.  Finally,  tests  of  student  proficiency  for  IMAT  training  include 
questions  that  require  problem  solving  and  understanding  causal  relationships  in  addition 
to  the  traditional  fact  recognition  and  procedural  process  items. 

Research  Background 

The  IMAT  system  integrates  several  areas  of  research  on  cognition  and  instruction, 
including,  graphical  techniques  to  promote  visualization  of  invisible  phenomena  in 
science  teaching,  elaborated  explanations,  contextualized  or  anchored  instruction,  and 
instructional  sequencing.  The  following  sections  briefly  summarize  portions  of  this 
work. 
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Scientific  Visualization.  Scientific  visualization  has  traditionally  been  used  by 
scientists  to  explore  phenomena  and  to  communicate  with  other  scientists  (Bryson,  1994). 
When  used  for  presentations,  researchers  select  data  sets,  transform  them  and  then  turn 
them  over  to  specialized  graphic  artists  to  develop  images  and  animation.  However,  the 
end  products  of  this  process  have  not  been  designed  for  laymen  or  students.  IMAT  aims 
to  bring  this  technology  into  specialized  technical  training. 

Research  support  for  scientific  visualization  as  a  training  strategy  comes  from  the 
literature  on  instructional  media.  Both  static  and  dynamic  graphic  displays  have  been 
shown  to  facilitate  teaching  of  scientific  concepts  (Baek  Layne,  1988;  Dwyer,  1972; 
Cropper,  1966;  Lunsdaine,  Sulzer,  &  Kopstein,  1961;  Rieber,  1990;  Rigney  &  Lutz, 
1975;  Park  &  Gittelman,  1992,  Wetzel,  Radtke,  &  Stem,  1994).  Levie  and  Lentz  (1982) 
in  a  meta-analysis  of  illustrated  text  studies  concluded  that  learning  and  retention  is 
facilitated  by  illustrations,  if  the  illustrations  are  directly  related  to  the  text.  Park  and 
Gittelman  (1992)  found  that  subjects  trained  with  dynamic  graphics  performed  better  on 
electronic  troubleshooting  problems  then  those  trained  with  static  displays.  White  (1984) 
used  animated  computer  graphics  to  successfully  teach  the  basic  principles  of  Newtonian 
laws  of  motion  and  force.  IMAT  employs  a  computer  based  graphical  interface  to 
conceptual  models  of  real  world  phenomena  to  deliver  both  static  and  dynamic  graphics 
in  a  traditional  classroom  environment. 

Elaborated  Explanations.  Providing  students  with  elaborated  explanations, 
analogies,  etc.  about  how  and  why  systems,  events,  and  phenomena  are  structured  and 
function  has  been  shown  to  facilitate  learning  and  retention.  Research  on  learning  skills 
and  learning  from  text  has  shown  that  elaborated  explanations  enhance  the  students’ 
mental  models  and  increase  retention  (Mayer,  1989;  Konoske  &  Ellis,  1991;  Smith  & 
Goodman,  1982).  In  a  series  of  studies  of  learning  from  scientific  text,  Mayer  (1989) 
found  that  providing  students  with  a  conceptual  model  increased  learning,  retention  and 
transfer.  The  conceptual  models  in  his  instruction  used  both  text  and  diagrams  to 
highlight  major  objects  and  actions  and  the  causal  relations  among  them.  That  is,  the 
models  focused  on  how  and  why  systems  work.  Smith  and  Goodman  (1982)  studied  the 
effects  of  providing  elaborated  instructions  on  learning  and  performing  a  procedural 
assembly  task  and  found  that  instructions  containing  functional  information  resulted  in 
fewer  errors.  Swezey,  Perez,  and  Allen  (1991),  in  a  study  on  transfer  of 
electromechanical  troubleshooting  skill,  found  that  some  level  of  generic  structure  and 
functional  knowledge  is  required  for  cross  domain  transfer.  The  IMAT  system  uses 
elaborated  explanations  throughout  the  instruction  to  (1)  clarify  complex  relationships 
such  as  those  among  water  temperature,  pressure  and  depth,  and  salinity;  (2)  provide 
comprehensive  feedback  for  practice  exercise;  and  (3)  describe  graphically  displayed 
examples. 

Contextualized  Instruction  and  Instructional  Sequencing.  Contextualized  or  job 
oriented  instruction  has  been  found  to  be  more  effective  in  learning,  retention,  and 
performance  than  topic  oriented  instruction  (Semb  &  Ellis,  1994;  Johnson,  1951; 
Goffard,  Heimstra,  Beecroft  &  Oppenshaw,  1960;  Shoemaker,  1960;  Steinemann, 
Harrigan,  &  VanMatre,  1967;  Cognition  &  Technology  Group  at  Vanderbilt,  1990; 
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Collins,  Brown,  &  Newman,  1989).  Further,  within  a  job  context,  mental  model 
development  is  facilitated  by  teaching  students  to  reason  about  events  and  phenomena 
that  involve  several  interrelated  variables.  Proper  sequencing  may  play  an  important  role 
in  cognitive  skill  development.  While  early  research  on  sequencing  showed  that  with 
simplified  or  isolated  tasks,  different  sequences  of  instructional  events  made  little 
difference,  more  recent  research  and  theory  suggests  that  for  complex  tasks,  sequencing 
strategies  may  have  significant  effects.  For  example,  Reigeluth  and  Stein  (1983)  argues 
that  beginning  instruction  with  a  condensed  “holistic”  overview  of  a  task  domain  leads  to 
better  learning  than  more  traditional  sequences,  which  teach  isolated  topics  first  and 
integrate  them  later.  More  recently,  extreme  “constructivist”  approaches  to  instruction 
(e.g.  Duffy  &  Jonassen,  1991)  argue  that  learners  should  “sink-or-swim”  in  a  fully 
elaborated  domain.  Merrill,  Li,  «&  Jones  (1990)  also  argue  for  a  holistic  approach  to 
teaching  complex  domains,  but  include  moderate  structure  and  sequencing 
recommendations  in  their  approach.  Drawing  from  Reigeluth  (1983),  IMAT  begins  with 
a  simplified  overview  of  target,  environment,  and  sensor  system  relationships  in  the 
context  of  the  jobs  and  tasks  performed  by  operators  and  tacticians.  This  context  is 
revisited  throughout  IMAT  to  reinforce  the  reality  that  students  are  learning  to  do  a  job 
not  memorize  a  list  of  topically  related  facts. 

AW  “A”  School—Acoustic  Analysis  for  Sensor  Operators 

In  each  of  these  areas,  little  experimental  work  has  been  done  on  the  extent  to  which 
the  findings  are  generalizable  to  instruction  delivered  using  simulation-  and  graphical- 
interface-based  training  technologies.  Furthermore,  there  are  almost  no  larger  efforts  that 
evaluate  the  integration  of  these  approaches  into  an  overall  strategy.  The  current  effort 
tests  the  hypothesis  that  the  IMAT  system,  which  represents  an  integrated  combination  of 
these  approaches,  offers  a  potent  learning  environment  for  promoting  acquisition  of  the 
complex  knowledge  and  skills  involved  in  sensor-system  operation.  Specifically,  this 
report  documents  the  implementation  and  evaluation  of  the  IMAT  system  in  the  acoustic 
analysis  portion  of  the  AW  “A”  school  in  Pensacola.  Although  IMAT  replaced  the 
existing  acoustic  analysis  lessons  in  the  “A”  curriculum,  the  basic  classroom 
configuration  and  environment  were  not  altered.  However,  the  IMAT  lessons 
represented  a  significant  change  firom  the  existing  instruction.  The  course  length  was 
increased  from  212  to  230  hours  and  there  were  extensive  changes  in  both  the  instructor 
and  student  guides,  which  were  redesigned  using  IMAT  criteria  to  provide  both 
elaborated  explanations  and  more  contextualization.  Furthermore,  the  IMAT  graphical 
interface  was  used  to  provide  the  classroom  instructor  with  both  static  and  dynamic 
displays  of  important  concepts  and  relationships  including,  low  frequency  analysis 
recording  (LOFAR)  displays,  submarine  operating  characteristics,  submarine  equipment 
acoustic  profiles,  and  target  analysis  and  classification.  All  of  these  changes  resulted  in  a 
course  that  emphasized  learning  cognitive  skills  and  complex  relationships  instead  of 
memorizing  factual  information,  which  was  the  focus  of  the  existing  unit. 
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Objective 


The  objective  of  this  effort  is  to  evaluate  the  application  of  the  IMAT  system  in  the 
AW  “A”  with  respect  to  performance  on  factual  and  cognitive  skill  test  items,  student 
motivation,  and  instmctional  design. 


Method 


Design 

The  AW  “A”  IMAT  application  was  evaluated  on  (1)  student  performance  on  two 
types  of  end  of  unit  test  items  (fact  items  and  cognitive  skill  items),  (2)  student 
motivation,  and  (3)  quality  of  instructional  design.  Students  trained  with  the  IMAT 
system  were  compared  with  students  trained  with  the  conventional  instruction  used  in  the 
“A”  school  prior  to  the  introduction  of  IMAT. 

Subjects 

The  subjects  were  22  students  who  had  completed  the  AW  “A”  school  acoustic 
analysis  lessons  prior  to  the  introduction  of  the  IMAT  system  (the  standard  instruction 
group),  and  59  students  who  were  taught  the  same  unit  with  IMAT  (the  IMAT  instruction 
group).  Although  ASVAB  scores  were  not  available  because  of  privacy  act 
considerations,  the  AW  “A”  school  has  ASVAB  entry  requirements  that  limit  the  range 
of  student  ability.  Given  random  entry  into  the  school,  the  student  samples  were 
considered  to  be  equivalent  in  ability. 

Instructional  Materials 

The  lesson  materials  for  both  groups  taught  acoustical  analysis  and  all  groups  were 
compared  on  identical  end  of  unit  test  items.  The  major  topics  addressed  were  LOFAR 
display  analysis,  submarine  drive  train  and  drive  train  component  analysis,  analysis  of 
mechanical  and  electrical  machinery  systems,  and  operational  modes. 

Standard  Instruction 

The  number  of  learning  objectives  for  the  standard  instruction  is  48,  which  are  taught 
in  212  hours  with  99  class  hours  and  113  lab  hours.  The  instruction  is  topic  oriented 
rather  than  teaching  the  knowledge  and  skills  in  a  job  context.  Neither  the  instructor 
guide  nor  the  student  guide  provide  many  opportunities  for  practice  in  preparation  for 
taking  the  unit  test. 

IMAT  Instruction 

The  number  of  learning  objectives  for  the  IMAT  instruction  is  159,  which  are  taught 
in  230  hours  with  146  class  hours  and  82  lab  hours.  The  instruction  is  presented  in  a  job 
context  with  practice  opportunities  for  each  objective. 
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End  of  Unit  Test 


After  completing  the  acoustic  analysis  lessons,  students  were  given  an  end  of  unit 
test,  which  included  items  that  could  be  used  to  compare  IMAT  and  standard  instruction. 
The  test  items  were  classified  into  one  of  two  categories  according  to  the  type  of 
information  tested  and  the  cognitive  processing  required  to  answer  the  question: 

Remember  fact.  The  student  must  recall  or  recognize  names,  definitions, 
steps  of  procedures,  formulas  an  terms  in  formulas,  labels  for  graphical  display, 
or  technical  terminology. 

Cognitive  skills.  The  student  must  perform  a  sequence  of  steps,  including 
making  decisions  and  judgments,  to  solve  a  problem,  calculate  or  determine  a 
value  or  relationship,  or  evaluate  a  scenario. 

The  test  items  used  for  the  evaluation  were  selected  from  a  pool  of  test  items 
developed  for  testing  acoustic  analysis  knowledge  and  skills.  All  test  items  in  the  pool 
were  reviewed  by  AW  “A”  course  instructor  personnel  and  by  subject  matter  experts 
assigned  to  the  IMAT  project.  Pool  items  that  were  rated  beyond  the  scope  of  the 
instruction  or  inappropriate  for  “A”  school  training  were  eliminated.  Test  items  selected 
for  the  evaluation  were  4  fact  items  and  81  cognitive  skill  items.  The  81  cognitive  skill 
items  were  divided  into  two  groups:  (1)  those  explicitly  taught  by  both  the  standard  and 
IMAT  instruction  (common  cognitive  skill  items)  (N  =  62)  and  (2)  those  topically  related 
to  the  standard  instruction  but  only  specifically  taught  by  the  IMAT  instruction  (IMAT 
cognitive  skill  items)  (N  =  19).  The  three  groups  of  items  (facts,  common  cognitive 
skills,  and  IMAT  cognitive  skills)  were  analyzed  separately. 

Motivational  Analyses 

For  the  motivational  analysis,  a  questionnaire  based  on  the  Attention-Relevance- 
Confidence-Satisfaction  (ARCS)  motivational  assessment  model  originated  by  Keller 
(1992)  was  developed.  The  34  item  questionnaire,  designed  for  application  to  technical 
training,  was  administered  to  all  of  the  IMAT  instruction  students.  Unfortunately,  the 
motivation  questionnaires  for  the  standard  instruction  students  were  either  not  collected 
or  were  lost  in  transit  and  could  not  be  analyzed.  In  a  previous  study  (Ellis  &  Parchman, 
1994),  the  ARCS  questionnaire  was  administered  to  76  AW  “A”  school  students  trained 
with  the  IMAT  system.  These  data  were  used  in  the  present  evaluation  for  comparison. 
The  ARCS  questionnaire  assess  four  motivational  characteristics;  attention,  confidence, 
relevance,  and  satisfaction.  The  attention  oriented  questions  assess  how  well  the  material 
captures  the  interest  of  the  learners  and  stimulates  their  curiosity  to  learn.  The  relevance 
questions  address  how  well  the  materials  meet  the  needs  and  goals  of  the  learner.  The 
confidence  questions  ask  the  students  to  report  on  their  beliefs  and  feelings  about  how 
well  they  will  succeed  and  how  much  they  can  control  their  own  success.  The 
satisfaction  questions  concern  students  feelings  of  reward  and  accomplishment  and  their 
enjoyment  of  the  materials.  The  four  scales  are  scored  from  1  to  5  with  1  indicating  a 
statement  about  the  course  is  “Not  true”  and  a  5  indicating  a  statement  is  “Very  true.”  A 
score  of  3  indicates  students  believe  a  statement  is  “Moderately  true.” 
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Instructional  Design  Evaluation 

The  Course  Evaluation  System  (CES)  (Ellis,  Knirk,  Taylor,  &  McDonald,  1993)  was 
used  to  compare  the  instructional  design  of  the  IMAT  and  standard  instruction  units.  The 
CES  assesses  the  consistency  (match)  among  learning  objectives,  test  items,  and  the 
instructional  presentation,  and  the  adequacy  of  the  instructional  presentation.  Although 
the  objectives  for  the  IMAT  and  standard  instruction  groups  were  different,  the  test  items 
were  identical.  Therefore,  the  CES  was  used  to  assess  the  consistency  among  the  test 
items  and  the  instructional  presentations  for  each  unit.  The  consistency  assessments  were 
then  compared  for  the  evaluation.  The  consistency  assessment  employed  the  same 
classification  scheme  for  test  items  used  for  the  end  of  unit  test. 

Results  and  Discussion 


Test  Analyses 

A  2  Groups  (standard  instruction  vs.  MAT  instruction)  by  3  Item  Type  (fact  vs. 
common  cognitive  skill  vs.  MAT  cognitive  skill)  analysis  of  variance  (ANOVA)  was 
performed  on  percent  correct  for  each  type  of  item  on  the  end  of  unit  test.  Table  1 
presents  the  means  for  this  analysis.  The  main  effects  group  and  the  item  type  by  group 
interaction  were  both  significant  (p  <  .01). 

Table  1 

Mean  Percent  Correct,  Standard  Deviations,  and  Effect  Sizes  for  Fact, 
Conunon  Cognitive  Skill,  and  IMAT  Cognitive  Skill  Test 
Items  for  IMAT  and  Standard  Instruction  Groups 


Facts 

N  =  4 

Common 

Cognitive 

Skills 

N=62 

IMAT 

Cognitive 

Skills 

N-19 

Total 
iV  =  85 

STG  standard 

instruction 

N=22 

76.1 

SD  =  .20 

72.1 

SD=.ll 

49.0 

SD  =  .13 

67.1 

SD  =  .10 

MAT 

instruction 

N=53 

81.1 

SD  =  .16 

86.7 

SD  =  .08 

82.5 

SD  =  .10 

85.5 

SD  =  .08 

Effect  Sizes 
(Pooled  SD) 

.29 

1.33 

1.77 

1.57 

The  finding  that  MAT  students  performed  better  on  all  three  types  of  test  items 
supports  the  hypothesis  that  the  combination  of  strategies  incorporated  in  the  MAT 
system  provides  an  effective  environment  for  learning  complex  knowledge  and  skills. 
The  most  important  result  for  the  hypothesis  is  that  MAT  instruction  students  performed 
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better  on  both  types  of  cognitive  skill  items  relative  to  fact  items  then  the  standard 
instruction  students.  That  is,  the  significant  groups  by  item  type  interaction  shows  that 
the  IMAT  system  differentially  increased  learning  and  performance  for  the  more 
complex  types  of  items.  This  was  especially  true  for  the  IMAT  cognitive  skill  items, 
which  reflect  the  addition  of  more  complex  objectives  to  the  IMAT  curriculum.  This 
result  is  exactly  what  IMAT  was  designed  to  accomplish  and  directly  addresses  the  post 
Cold  War  training  requirements  for  sensor  operators. 

Instructional  Design  Analyses 

The  significant  differences  among  the  groups  on  all  three  types  of  test  items  can,  in 
part,  be  explained  by  comparing  the  degree  to  which  some  of  the  fundamental 
instructional  principles  underlying  the  IMAT  system  were  applied  in  each  unit.  The 
results  of  the  CES  analysis  reveals  these  differences.  For  example,  the  total  number  of 
objectives  for  the  IMAT  lessons  was  159  versus  48  for  the  standard  instruction  lessons. 
This  resulted  in  part  from  the  IMAT  lessons  being  completely  redesigned.  The  IMAT 
redesign  reflects  a  thorough  front-end  analysis  of  the  knowledge  and  skills  that  are 
required  for  proficiency  and  a  concern  for  teaching  the  cognitive  concepts  and  skills 
needed  to  perform  acoustic  analysis  in  a  post  Cold  War  world  environment.  These 
differences  in  number  of  objectives,  depth  of  analysis,  and  training  emphasis  probably  all 
worked  to  improve  the  performance  of  IMAT  students..  However,  the  relatively  small 
differences  in  unit  length  (230  hours  for  IMAT  vs.  212  hours  for  the  standard  instruction) 
probably  had  little  impact  on  the  group  differences.  In  fact,  there  was  more  laboratory 
time  in  the  standard  instruction  (113  hours)  then  in  the  IMAT  instruction  (82  hours),  and 
since  laboratory  time  was  used  to  practice  many  of  the  cognitive  skills,  this  should  have 
aided  standard  instruction  students.  Further,  the  common  cognitive  skill  items  were 
taught  in  both  courses  and  used  similar  amounts  of  instructional  time.  (See  Table  2  for 
percent  of  instructional  components  for  each  group.)  The  group  difference  on  IMAT 
cognitive  skills  is  only  difference  that  can  be  attributed  to  more  instructional  time  for 
IMAT  students  because  those  skills  were  not  explicitly  taught  in  the  standard  instruction. 

The  CES  evaluation  of  the  consistency  among  instructor  and  students  guides  and  the 
test  items  also  reveals  differences  among  the  two  groups  that  may  account  for  the 
performance  differences.  Table  2  shows  percent  of  instructional  components  that  are 
present  in  the  instructor  and  student  guides  for  each  type  of  test  item  for  each  unit.  For 
example,  if  there  were  10  fact  test  items,  each  item  would  require  a  statement  and  an 
opportunity  to  practice  remembering  to  be  present  somewhere  in  the  instructor  or  student 
guides.  If  there  were  only  five  statements  and  three  practice  questions  for  the  10  fact 
items,  the  percentages  in  the  Statement  and  Practice  Remembering  w/Feedback  columns 
would  be  50  percent  and  30  percent,  respectively.  (Note  that  for  fact  items,  the  example 
and  Practice  Using  components  are  not  required.)  In  the  IMAT  lessons  all  required 
components  are  present,  while  components  are  present  to  a  lesser  degree  for  the  standard 
instruction  lessons.  In  the  standard  instruction  the  practice  component  for  facts  is 
missing  50  percent  of  the  time.  The  absence  of  practice  and  other  required  instructional 
components  for  specific  test  items  in  the  standard  instruction  may  account  for  some  of 
the  performance  differences  between  the  groups. 
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Table  2 


Percent  of  Each  Required  Presentation  Component  that  is 
Present  for  Each  Type  of  Test  Item  for  the 
IMAT  and  Standard  Instruction  Units 


Req 

uired  Presentation  Components 

Test  Item  Type  and 
Instractional  Unit 

Statement 

Practice 

Remembering 

w/Feedback 

Practice 

Using 

w/Feedback 

Standard  Instruction 

Fact  (4  Items) 

75.0 

50.0 

NA 

NA 

Common  Cognitive 
Skill  (62  Items) 

95.4 

NA 

93.9 

95.4 

IMAT 

Fact  (4  Items) 

100.0 

100.0 

NA 

NA 

Common  Cognitive 
Skill  (62  Items) 

100.0 

NA 

100.0 

100.0 

Motivational  Analysis 

The  ARCS  questionnaire  assesses  four  motivational  characteristics;  attention, 
confidence,  relevance,  and  satisfaction.  Each  characteristic  is  rated  from  one  to  five;  not 
true  to  very  true,  respectively.  The  mean  scores  for  each  scale  for  the  IMAT  instruction 
are:  Attention  =  3.14,  Confidence  =  3.40,  Relevance  =  3.40,  and  Satisfaction  =  3.33  (N  = 
53).  These  scores  are  not  significantly  different  from  the  ARCS  scores  for  an  IMAT 
oceanography  lesson  taught  at  the  AW  “A”  school  which  were  collected  in  previous 
studies  (Ellis  &  Parchman,  1994;  Wetzel-Smith,  S.  K.,  Ellis,  J.  A.,  Reynolds,  A.  M.,  & 
Wulfeck  W.  H.,  1995).  Those  scores  were  Attention  =  3.43,  Confidence  =  3.35, 
Relevance  =  3.76,  and  Satisfaction  =  3.49.  As  reported  in  Wetzel-Smith,  S.  K.,  Ellis,  J. 
A.,  Reynolds,  A.  M.,  &  Wulfeck  W.  H.  (1995),  motivation  scores  in  the  ranges  obtained 
in  the  present  IMAT  instruction  lessons  and  the  previous  IMAT  oceanography  unit  are 
significantly  higher  than  scores  obtained  for  traditional  classroom  instruction  an 
introductory  electricity  course. 


Conclusions 

Several  conclusions  can  be  drawn  from  this  evaluation.  First,  research  on  cognition 
and  instruction  and  technological  advances  in  scientific  visualization  can  be  integrated 
and  applied  in  real  world  training  to  produce  substantial  gains  in  performance  and 
student  motivation.  Second,  the  IMAT  system  has  achieved  its  intended  design  goals  by 
effectively  teaching  complex  knowledge  and  cognitive  skills.  Third,  the  IMAT  system 
emphasis  on  inclusion  of  required  instructional  components,  especially  practice 
opportunities,  contributed  significantly  to  the  observed  performance  improvements. 
Overall,  the  results  show  that  IMAT  is  a  highly  effective  training  system  that  offers  a 
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viable  solution  for  many  of  the  training  requirements  and  challenges  faced  by  the  four 
ASW  communities  (surface,  subsurface,  air,  and  surveillance)  in  the  post  Cold  War 
world. 
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